Welcome to ECroPS’s documentation!¶
Indices and tables¶
Packages code documentation:
ECroPS Manual¶
Ecrops Manual
This is the manual of the ECroPS model platform. The present document refers to ECroPS version 1.9.0 (released July 2025). More information regarding the agronomic description of the models and the parallelization of the model can be found in the ECroPS technical report
What is ECroPS¶
ECroPS (Engine for Crop Parallelizable Simulations) is a software platform to build and run agronomic models. It was developed by JRC, Unit D5.
Implemented models¶
The first agronomical model implemented in the platform is the Wofost model (Wofost 7.1, from PCSE version 5.3.1). See Wofost documentation here: https://wofost.readthedocs.io/en/latest/# For writing the Wofost implementation, we started from the PCSE software developed by Wageningen University and Research. See PCSE documentation here: https://pcse.readthedocs.io/en/develop/ . PCSE contains a full python implementation of Wofost. We extracted a part of the Wofost modules and we re-coded them in the ECroPS platform, saving most of the algorithmic part of the code, but changing many aspects like the way variables are stored and the simulation cycle. At the end of the conversion, a validation test was made to verify the correctness of the new implementation in terms of simulation results.
It was then implemented the WARM model (version 1.9), originally developed by professor Roberto Confalonieri (Cassandra Lab at University of Milan). WARM was written in C# in the Bioma platform, so the code was re-written from scratch. In addition, in this case it was performed a validation test phase to verify the correctness of the new version. See WARM documentation here: http://cassandralab.com
In Autumn 2023 the FROSTOL and Ritchie model for frostkill calculation and the LINGRA model for grassland simulations were implemented in the platform.
Published software¶
Since October 2022 eCroPS is published on the JRC Github page (version 1.5), at this address: https://github.com/ec-jrc/ecrops. It is published under license EUPL 1.2. The published software covers the core engine and the Wofost/WARM implementations (package ecrops), plus an example of python console program to launch a simulation (folder EcropsWofostExampleConsole) as described in section ‘ECroPS model direct run’ of the present manual.
Technical implementation of the platform¶
In this platform, we describe the agronomical model as a succession of steps to be executed. Each step represents the minimal algorithmic part of the model. Steps are daily based, and they are executed in a cycle for a period between a start and an end date, provided in the configuration. The engine of the ECroPS platform is designed to work considering model’s steps, using configuration files to define the steps to execute and execution order. The steps are executed one after the other for the desired time iterations. The platform has no limitations for the number of steps or for the execution order.
More than that, the platform offers the possibility to load dynamically the steps, to define variables in dynamic manner, through configuration files (all configuration files are XML).
The model variables are defined inside a “status” variable, which is generated by the initialize method and then updated and returned by every step in the model.
Workflow of the ECroPS engine¶
The ModelEngine class is used to run the ECroPS model. When an instance of this class is created, it reads the configuration from a provided configuration file. This file defines the structure of the model execution in terms of steps to be executed, input data, and output variables. The class has several properties that define the driving variables, the instructions to initialize the status variables, the steps to run, and the output variables.
The simulation cycle is divided into three parts: the initialize method checks that the required driving variables are present and initializes the status variables with the provided input data; the executeStep method performs the simulation by calling the steps in the correct order for each day in the cycle; and the finalize method returns the calculated output variables.
The status of the model and its variables are stored in a “status” variable, which is modified by each model method. This variable is created by the initialize method and returned to the caller. It is the caller’s responsibility to manage the status variable and pass it to the executeStep and finalize methods as needed. This approach allows the caller to have full control over the status variable and perform custom tasks.
To use the ModelEngine class, a caller can follow the generic workflow:
Create a ModelEngine instance with the config file:
model = ModelEngine(config_file)(*)Initialize the model with the required input variables:
status = model.initialize(timedependantvariables, drivingVariables, allparameters, first_day, simulation_start_day, simulation_end_day)For each simulation cycle, run the executeStep method with the current status:
status = model.executeStep(status)After the last simulation cycle, run the finalize method to get the output:
result = model.finalize(status)
(*) Note: from version 1.7.2 it is possible to create an instance of ModelEngine passing a string containing the configuration, instead of the configuration file.
In this case the usage of the constructor is: model = ModelEngine(config_string, file_mode=False). The default of file_mode is True.
The ModelEngine class can be used in different “run modes” that define which steps to execute. This allows the same model to be run in slightly different ways.
The configuration file contains:
the instructions to initialize the status variables (property initVariables)
(optional) the instruction to initialize the driving variables
the steps to run (property Workflows)
the list of variables to return as output (property Workflows)
The simulation cycle is divided in 3 parts:
the initialize method initializes the status variable from the property initVariables and the provided input data
the executeStep method performs the simulation by calling the steps in the correct order for every day in the cycle. For every day between simulation_start_day and simulation_end_day, if requested, the method populates a dailydetails array with the daily value of the output variables
the finalize method returns an array with output variables calculated after the last time interval executed
The model’s status and variables are all contained in a variable called “status”, which is modified by every model method. The status variable is NOT contained in the ModelEngine class: it is created by the initialize method and returned to the caller. The status is a required input of both executeStep and finalize methods. Every call to the executeStep method updates the status with the variable values calculated inside the step. It is duty of the caller to manage the status variable properly. This approach allow the caller to have a full control of the status variable to perform specific tasks.
Here the generic workflow of the caller:
model = ModelEngine(config_file)
status=model.initialize(timedependantvariables, timeDependantVariableColumn, drivingVariables, allparameters, first_day, simulation_start_day, simulation_end_day)
for every step of the simulation cycle:
status=model.executeStep(status)
result=model.finalize(status)
The input data of the model, passed inside the ‘initialize’ method, can be distinguished in three types and so three distinct arguments are passed to the method:
argument ‘timedependantvariables’: the time dependant variables are all the input variables that change value according to the time step, typically the weather data.
argument ‘allparameters’: dictionary containing all the parameters of the model. The keys of the dictionary are the crop identifiers
argument ‘drivingVariables’: the driving variables are all the variables that defines the configuration of the simulation to run, for example the crop to be run, the years to be run, the latitude and longitude of the simulated location, the soil properties of the location. In other words, the driving variables contains all the input data except the time dependant variables and the model parameters.
- The reading of the workflow is done in the constructor method of the ModelEngine class. Example:
model = ModelEngine(“my_workflow_file.xml”)
The workflow can be executed on more than one “run mode”. A “run mode” defines the steps to execute and can be used when the same model should be executed in slightly different ways, as, for example, adding or not adding a specific step. The reading of the workflow file is done at the beginning of the workflow, since that files defines the structure of the model execution in terms of steps to execute, input data and output variables. See section “Description of the workflow configuration file”.
The “initialize” method of the ECroPS engine initialize the “status” variable, by using all the inputs defined in the “Init” section of the workflow file. For example, if in the “Init” section is present this line
<Variable name="LAT" source="drivingVariables['LAT']" />
then the engine executes this line of code
status.LAT=drivingVariables['LAT']
so it instantiates a new variable called “LAT” in “status” and set its value to the value get by executing the instruction
drivingVariables['LAT']
where drivingVariables is one of the arguments of the “initialize” method.
Moreover, the “initialize” method checks that all the drivingVariables defined in the configuration file are set. In case one is missing an exception is thrown. The drivingVariables dictionary contains all the driving variables (e.g. the year and the crop to simulate, the soil data,…) Finally, the “initialize” method allow to pass to the engine the time dependant variables (usually the daily weather data). The time dependant variables are packed as a 2-dimensional array, called ‘timedependantvariables’, where the first dimension specifies the variable (e.g. the weather variable), the second dimension specifies the time step (e.g. the day). The names of the variables of the first dimension are specified in the property ‘timeDependantVariableColumn’: this is a dictionary containing the names of the variables and their relative column number in table ‘timedependantvariables’. Example:
timeDependantVariableColumn = {'TEMP_MAX': 0, 'TEMP_MIN': 1, 'IRRAD': 2, 'RAIN': 3, 'WIND': 4, 'RH': 5, 'E0': 6, 'ES0': 7, 'ET0': 8}
In this example, the temperature max (TEMP_MAX) is in the first column of the table timedependantvariables.
The “executeStep” method of the ECroPS engine runs all steps for specific run mode, in the defined order, for every value of status.day property when
status.simulation_start_day <= status.day <= status.simulation_end_day
status.day is incremented by 1 day at the end of the method.
The first value of status.day is the argument first_day of method initialize.
For every step, the “status” variable is set as an input of the step, is updated by the step and returned to the engine. Here the “executeStep” method workflow:
The current time interval counter (variable status.day) is set equal first_day
if current time interval counter is between “simulation_start_day” and “simulation_end_day”:
if current time interval counter is equal to “simulation_start_day”:
for each step, call step’s “setparameters” method
for each step, call step’s “initialize” method
if current time interval counter is greater than “simulation_start_day”:
for each step, call step’s “integrate” method
for each step, call step’s “runstep” method
if variable “ReturnDailyDetails” is True, add the current time interval output variables to the “status.dailydetails” variable
add 1 day to the current time interval counter
The “finalize” method of the ECroPS engine generates an array with output variables calculated after the last time interval executed and returns it. To do so, it uses the output variables definition read from the workflow file. It also returns the daily details object if ReturnDailyDetails or ReturnDekadalDetails are True. Finally, if PrintDailyDetails or PrintDailyDetailsToFiles is True, it prints the content of the daily details object to the console output or to the specified file (PrintDailyDetails_OutputFile). So it returns a tuple containing two values: 1) the summary output array. It is a 1D array having size = NUM_OUTPUT_VARIABLES 2) the daily details dictionary (if ReturnDailyDetails and ReturnDekadalDetails are False, the returned daily details object is None). The dictionary contains one item for each output variable: the key is the variable name, the value is a list: the list contains one item per simulated day. Each item of the list is the value of the output variable at that day.
Description of the methods of a step¶
Each step must have a well-defined structure, to be properly managed by the platform. In particular, each step need to be implemented like a class with specific methods, like described below:
initialize – has the role to initialize the step data, for example the status variables used in the step (it is called only before executing the step for the first time)
setparameters – prepare the parameters necessary for the step run (it is called only before executing the step for the first time)
integrate – has the role to merge the values of the previous time interval before the calculation of the current time interval step operations.
runstep – has the role to execute all the step’s operations
getparameterslist – returns the list of the parameters managed by the step
getinputslist (new from version 1.7.0) – returns the list of the inputs needed by the step
getoutputslist (new from version 1.7.0) – returns the list of the outputs calculated by the step
The class can implement any other methods, but the previous list of methods are mandatory for the correct run of the engine.
(new from version 1.7.0) The step class should be an implementation of the ecrops.Step class, defined in the ecrops package. ecrops.Step is an abstract class that defines, as abstract methods, the methods listed above.
Each implementation of methods initialize, setparameters, integrate and runstep should accept as argument the status variable and return the updated status variable.
The trivial example is:
def initialize(self,status):
#do something on the status
return status
Description of the workflow configuration file¶
The main configuration file of the platform is the workflow configuration XML file. Inside the file, it is possible to configure:
Initial variables: the input variables of the model simulation
Driving variables: (optional) the definition of the driving variables set in the ‘drivingVariables’ argument of the initialize method of the Engine. The driving variables could be used to define the values of the initial variables
Steps and their order: the logics to be simulated
Output variables: the output variables to be returned to the caller
In the next paragraphs, it is presented the configuration that can be present inside the XML workflow configuration file.
The section Init (defined by tag Init) contains the definition of the variables that will be initialized at the start of the simulation. These are the attributes of the tag Variable:
name: describes the name of the variable to be created. The variable will be created as attribute of the status property using the provided name. For example is name is ‘myvar’, it will be created a variable status.myvar = value provided by source attribute
source: describes the way to retrieve the value for the variable. The source represents the way the Engine can get the variable value from the input data. Input data include arguments drivingVariables, timedependantvariables, allparameters, first_day, simulation_start_day, simulation_end_day. Alternatively, the value of the variable can be directly written in the workflow file. Any valid python expression is valid as a value.
There is no limits for the number of tag Variable inside the Init section.
Examples of input variable definitions:
<Variable name="V" source="’my string’" />
In the sample above, the value of the variable is equal to the constant ‘my string’.
<Variable name="V" source="57" />
In the sample above, the value of the variable is equal to the constant 57.
<Variable name="V" source="first_day" />
In the sample above, the value of the variable is equal to argument ‘first_day’.
<Variable name="V" source="drivingVariables['LAT']" />
In the sample above, the value of the variable is retrieved from the element ‘LAT’ from the dictionary with the name drivingVariables).
<Variable name="V" source="0 if drivingVariables['DEPTH'] <= 0 else drivingVariables['DEPTH']" />
In the sample above, the value of the variable is 0 if the value of the element ‘DEPTH’ from the drivingVariables is <=0, otherwise it is equal to the value of ‘DEPTH’).
The section DrivingVariables (defined by tag DrivingVariables) contains the definition of the driving variables that are passed to the model inside the Initialize method of the ecrops ModelEngine. The driving variables are all the variables that defines the configuration of the simulation to run, for example the crop to be run, the years to be run, the latitude and longitude of the simulated location, the soil properties of the location,… In practice, the driving variables contains all the input data except the time dependant variables and the model parameters.
Example:
<DrivingVariables>
<DrivingVariable name="YEAR" description="Year" unitofmeasure="" type="numeric" />
<DrivingVariable name="DURATION" description="Number of days to run" unitofmeasure="" type="numeric" />
<DrivingVariable name="Crop" description="Crop" unitofmeasure="" type="numeric" />
<DrivingVariable name="LAT" description="Latitude" unitofmeasure="degrees" type="numeric" />
....
The DrivingVariable section is optional. Driving variables do not need to be explicitly listed in the section. Adding this explicit list to the configuration file allows the engine to verify all the necessary variables where passed to the engine inside the Initialize method. If the section is not present in the workflow configuration file, the check is not performed. Otherwise, if the section is present, the engine, inside the Initialize method, checks that all the declared driving variables are present in the ‘drivingVariable’ dictionary passed as argument of the Initialize method. If a variable is not present, an exception is thrown and the model does not proceed.
The driving variables can be accessed using the format
`drivingVariables['var_name']`
The driving variables can be used to initialize the Variables of the Init section, as in the example below:
<Init>
<Variable name="LAT" env="locals" source="drivingVariables['LAT']" />
....
These are the attributes of the tag DrivingVariable:
name: the name of the driving variable.
description: the description of the driving variable.
unitofmeasure: the unit of measure of the driving variable.
type: the type of the driving variable. Could be:
numeric
boolean
string
file: the path to a file containing some values
There is no limits for the number of tag DrivingVariable inside the DrivingVariables section.
The section Workflow (defined by the node Workflow) allows to configure the steps for a workflow and the output variables of the workflow. This tag must have two attributes:
name: specifies the workflow name
run: gives the possibility to switch off/on the execution of a workflow. Two values are allowed for this attribute:
ON: workflow will be executed
OFF: workflow will be ignored
The subsection Steps (child of the Workflow node) contains the (unlimited) list of the steps of the model. Each step is identified by a Step node, like follow:
<Step>ECroPS.co2effect.LinkCo2DataToAssimilation|LinkCo2DataToAssimilation</Step>
The node value contains two elements, separated by character ‘|’:
the reference to the file that contains the class that implements the step, represented by its python path including the package name
the name of the class that implements the step
The subsection Output (child of the Workflow node) contains the (unlimited) list of the output variables of the model. Different workflows can return a different set of outputs, and this is the reason why this sub-section is defined under the workflow node. Each output variable is identified by a Variable node, like follow:
<Variable source="status.states.DVS" name="POT_DVS" description="Potential DVS "/>
<Variable source="status.vernalisation.DOV.timetuple().tm_yday" name="POT_JDOV" description="Potential Julian day of vernalization end "/>
The Variable node must have three attributes:
source defines the variable name or a method to retrieve the value for the output variable starting from the model’s internal status. A variable to be extracted as output should belong to property ‘status’, so the source should start with ‘status’. E.g. ‘status.weather.RAIN’’
name identifies the output variable
description textual description of the variable
Dynamic classes loading¶
As described in the previous paragraphs, the step configuration (see the Step tag) allows to define a complete path for the python class to run: this means it is possible to specify the physical path and the class name that implements the step.
<Step>ECroPS.co2effect.LinkCo2DataToAssimilation|LinkCo2DataToAssimilation</Step>
Considering our example, the application searches the file LinkCo2DataToAssimilation.py into the path /ECroPS/co2effect. Inside the file, it should be present the class LinkCo2DataToAssimilation. The content of the tag Step must follow the convention:
<Step> path of the python file |name of the class that implement the step</Step>
Daily details managements¶
Besides the final output variables returned by finalize method, the model engine can also fill and return a ‘daily details’ object, where the engine saves the output variables at the end of every simulation time interval. So that the caller can save and analyse the behaviour of each output variable during the simulation. The daily details is particularly useful when analysing a single simulation output. The array can become huge, so it is recommended to disable it for batch simulations. To enable the ‘daily details array’, the Boolean property ‘ReturnDailyDetails’ of ModelEngine should be set to True. To retrieve the daily details array at the end of the simulation, it is sufficient to read property status.dailydetails.
Another property, called ‘PrintDailyDetails’, manages the print of the daily details. When set to true, the ModelEngine prints the daily details directly to the output console (standard output). To print the daily details to a file, the user should set the boolean property ‘PrintDailyDetailsToFile’ to true and set the property ‘PrintDailyDetails_OutputFile’ to the desired output file path.
Example of usage (returning but not printing daily details):
#initialize model
w = ModelEngine("Workflow.xml")
#return daily details
w.ReturnDailyDetails = True
# do not print automatically daily details
w.PrintDailyDetails = False
#....
#execute model
#....
#read the daily details
details=status.dailydetails
Example of usage (returning and printing daily details to a file):
#initialize model
w = ModelEngine("Workflow.xml")
#return daily details
w.ReturnDailyDetails = True
# print automatically daily details to file 'output.csv'
w.PrintDailyDetails = True
w.PrintDailyDetailsToFile = True
w.PrintDailyDetails_OutputFile = 'output.csv'
#....
#execute model
#....
#read the daily details
details=status.dailydetails
The daily details object is a dictionary. The dictionary contains one item for each output variable: the key is the variable name, the value is a list: the list contains one item per simulated day. Each item of the list is the value of the output variable at that day.
10-days details managements¶
Similarly to what explained in daily details management paragraph, it is possible to configure the model to return the details of the simulations at the end of every ‘dekad’: a ‘dekad’ is a period of circa 10 days defined as the first 10 days of the month, the second 10 days of the month, and the last days of the month starting from day 21.
1 - 10 first dekad of the month
11 - 20 second dekad of the month
21 - end-of-month third dekad of the month
As for the daily details, the resulting array can become huge, so it is recommended to disable it for batch simulations. To enable the ‘dekadal details array’, the Boolean property ‘ReturnDekadalDetails’ of ModelEngine should be set to True. To retrieve the dekad details array at the end of the simulation, it is sufficient to read property status.dailydetails. In this case the status.dailydetails object will contain only values for the 10th , the 20th and the last day of the month. If both ‘ReturnDekadalDetails’ and ‘ReturnDailyDetails’ are set to true, the daily results will be returned.
To print the dekad values, it should be used the property ‘PrintDailyDetails’ already described in the daily details managements paragraph. Also, it is possible to use the properties ‘PrintDailyDetailsToFile’ and ‘PrintDailyDetails_OutputFile’ as described in the previous paragraph to plot dedak output to file.
The dekadal details object is a dictionary, having a structure similar to the daily details object. The dictionary contains one item for each output variable: the key is the variable name, the value is a list: the list contains one item per simulated dekad. Each item of the list is the value of the output variable at the end of that dekad.
How to add a step into a model workflow¶
To add an existing step in a model workflow, the user should add the step definition in the XML workflow file, inside the ‘Workflow’ section, in the desired position. The user should ensure all the input required by the step are provided either by the previous steps, or are defined as variables in the Init section.
How to add a new step to an existing model¶
This is the procedure for adding a new step into an existing model:
Create a python class (like described in paragraph “Description of the methods of a step”) and place this new class into a valid python project. The class could be added directly to the ecrops package, but it is not mandatory.
Put the line of the new step into the desired position (like described in paragraph “Description of the workflow configuration file”) inside the workflow configuration file of the model. If the class is in a new python package, that package should be imported in the calling python code.
Example: the user creates a new step called ‘MyAlgorithm’ in a python package called “mypackage”. The class code, defined in file ‘MyAlgorithm.py’, is the following.
from ecrops.Step import Step
class MyAlgorithm(Step):
"""MyAlgorithm ecrops step"""
def getparameterslist(self):
return {}
def getinputslist(self):
return {}
def getoutputslist(self):
return {}
def setparameters(self, status):
return status
def initialize(self, status):
return status
def runstep(self, status):
return status
def integrate(self, status):
return status
The project structure is:
–> mypackage
——> __init__.py
——> MyAlgorithm.py
The project should be built as a valid package, called ‘mypackage’. The package should be imported by the python script that launches the model simulation. To include the MyAlgorithm step in the workflow configuration file, the user should add this line:
<Step>mypackage.MyAlgorithm|MyAlgorithm</Step>
If the new step is using some input variable not already present in the model, it is necessary to define it into the workflow configuration file (see the paragraph “Description of the workflow configuration file” – initial variables section).
(new from version 1.7.0) If the step does not extend class ecrops.Step an error will be triggered at the loading of the workflow.
Unit tests of the Step¶
(new from version 1.7.1) Since version 1.7.1, the ecrops package contains a method to run the unit tests of a step. The unit tests allow to verify the correctness of the step implementation by running the step for one time step (usually a day) and checking the outputs of the calculation. We provide testsets containing the inputs and the parameters to feed the step calculation and the expected outputs. The test sets are saved in form of XML files having this structure:
- ``<?xml version=’1.0’ encoding=’UTF-8’?>
- <UnitTest Step=”step name” StepModule=” step module “>
- <TestSets>
- <TestSet Description=”Description of the test set”>
- <Parameters>
…parameters of the step
</Parameters> <Inputs>
… inputs needed by the step
</Inputs> <Outputs>
… expected outputs
</Outputs>
- </TestSet>
… other test sets
<TestSet> </TestSet>
</TestSets>
</UnitTest>``
Below, as an example, we show a test set for the ActualRue Warm model step. Others xml are provided under folder ‘ecrops/unit_tests’.
- ``<UnitTest Step=”ActualRue” StepModule=”ecrops.FPWarm.ActualRue”>
- <TestSets>
- <TestSet Description=””>
- <Parameters>
<Parameter Name=”MaximumRadiationUseEfficiency” Value=”2.79902251918455”/>
</Parameters> <Inputs>
<Input Name=”status.rates.RUESaturationEffectRate” Value=”1”/> <Input Name=”status.rates.RUESenescenceEffectRate” Value=”1”/> <Input Name=”status.rates.RUETemperatureEffectRate” Value=”0.9360085347278598”/> <Input Name=”status.states.DevelopmentStageCode” Value=”1.3975263194033973”/> <Input Name=”status.UseSaturation” Value=”True”/> <Input Name=”status.UseSenescence” Value=”True”/> <Input Name=”status.UseTemperature” Value=”True”/> <Input Name=”status.UseCO2” Value=”False”/> <Input Name=”status.auxiliary.GrowthRatioCO2” Value=”1”/> <Input Name=”status.states.RUEActual” Value=”50”/>
</Inputs> <Outputs>
<Output Name=”status.rates.RUEActualRate” Value=”2.62”/> <Output Name=”status.states.RUEActual” Value=”52.62”/>
</Outputs>
</TestSet>
</TestSets>
</UnitTest>``
The provided xml file contains one or more test sets. A test set is composed by the inputs and the parameters needed to run the step for a particular time step (usually 1 day). The test set contains also the expected output variables values at the end of the execution of the time step.
To run the unit test, the user should call the method run_test of class ecrops.unit_tests.GenericEcropsStepUnitTest.
This method:
creates an instance of the step
sets the inputs and the parameters in the step
calls the step’s setparameters method
calls the step’s initialize method
calls the step’s integrate method
calls the step’s runstep method
checks all the outputs variables, by comparing the calculated values and the expected values. If one value does not match, an AssertionError is raised.
The method returns the “Ok” string if all the tests succeeded, otherwise an AssertionError is raised.
Note: in the check phase, not all the possible output types are managed. At the moment the check is performed on:
strings
numbers (by using a margin of 0.01, to avoid approximation issues)
dates (datetime.datetime)
In this snippet, we show how to launch the unit test for step CeresRitchie:
- ``t = GenericEcropsStepUnitTest()
t.run_test(‘frostkillCeresRitchie_unittest_dataset_example.xml’)``
Under folder ‘ecrops/unit_tests/blank_unittests’ we provided blank (=without values) templates to create test sets for all the implemented steps.
The user can create other blank templates by using the tool ecrops.unit_tests.UnitTestXMLFilesGenerator.
In this snippet, we show how to create the blank unit test for step ecrops.waterbalance.LayeredWaterBalance.WaterbalanceLayered :
g = UnitTestXMLFilesGenerator()
g.generateXml('ecrops.waterbalance.LayeredWaterBalance', 'WaterbalanceLayered')
SerializeModelInputConfiguration and DeSerializeModelInputConfiguration methods¶
Since version 1.7.2 the ModelEngine has two methods SerializeModelInputConfiguration and DeSerializeModelInputConfiguration to serialize and deserialize the configuration of the model inputs.
They use the pickle serialization library.
These methods are useful for save configuration of the inputs and reload them later, for example to make reproducibility tests.
SerializeModelInputConfiguration serializes the configuration of the model, passed as argument, including:
the xml configuration file
all the arguments of the ‘initialize’ method:
timedependantvariables
timeDependantVariableColumn
drivingVariables
allparameters,
first_day
simulation_start_day
simulation_end_day
and returns the pickle serialized string.
DeSerializeModelInputConfiguration deserializes the model starting from a pickle serialized string returned by method SerializeModelInputConfiguration.
It returns an instance of ModelEngine, on which the ‘initialize’ method was already called.
In detailts, it returns a list of tuples, one for each runmode defined in the configuration file, and active (run=”ON”). Every tuple contains:
the runmode name
the instance of a ModelEngine already initialized
the status variable after the initialization
Vectorial simulations: vecrops package and Vectorial Steps¶
From version 1.7.2 we introduced the possibility to run Vectorial steps.
This requires the installation of the vecrops package. Vectorial steps are implementations of class vecrops.VectorialStep, which in turns it is a class derived from ecrops.Step.
Vectorial steps are run by using the vecrops.VectorialModelEngine, instead of ecrops.ModelEngine. vecrops.VectorialModelEngine is a subclass of ecrops.ModelEngine: It inherits all the methods from its parent, except executeStep and finalize. These two methods are reimplemented since they should manage vectorial input and output data.
The “executeStep” method of the ECroPS engine runs all steps for specific run mode, in the defined order, for every value of status.doy property when
status.simulation_start_day <= status.doy <= status.simulation_end_day
status.doy is incremented by 1 at the end of the method.
The first value of status.doy is the argument first_day of method initialize.
For every step, the “status” variable is set as an input of the step, is updated by the step and returned to the engine. The “executeStep” method workflow is identical to that one of the scalar case:
The current time interval counter (variable status.doy) is set equal first_day
if current time interval counter is between “simulation_start_day” and “simulation_end_day”:
if current time interval counter is equal to “simulation_start_day”:
for each step, call step’s “setparameters” method
for each step, call step’s “initialize” method
if current time interval counter is greater than “simulation_start_day”:
for each step, call step’s “integrate” method
for each step, call step’s “runstep” method
if variable “ReturnDailyDetails” is True, add the current time interval output variables to the “status.dailydetails” variable
add 1 to the current time interval counter
The “finalize” method of the vectorial engine generates a 2D array with output variables calculated after the last time interval executed and returns it. To do so, it uses the output variables definition read from the workflow file. It also returns the daily details object if ReturnDailyDetails or ReturnDekadalDetails are True. Finally, if PrintDailyDetails or PrintDailyDetailsToFiles is True, it prints the content of the daily details object to the console output or to the specified file (PrintDailyDetails_OutputFile). So it returns a tuple containing two values: 1) the summary output array: the array containing the values of the output variables, in the same order they are returned by getOutputVariables and getOutputVariablesNames methods.The array has size {NUM_LOCATIONS x NUM_OUT_VARIABLES) 2) the daily details dictionary. The dictionary contains one item for each output variable: the key is the variable name, the value is a list: the list contains one item per simulated day. Each item of the list is the value of the output variable at that day.
The workflow of vecrops.VectorialModelEngine is identical of the workflow of ecrops.ModelEngine, described above:
Create a VectorialModelEngine instance with the config file:
model = VectorialModelEngine(config_file)Initialize the model with the required input variables:
status = model.initialize(timedependantvariables, drivingVariables, allparameters, first_day, simulation_start_day, simulation_end_day)For each simulation cycle, run the executeStep method with the current status:
status = model.executeStep(status)After the last simulation cycle, run the finalize method to get the output:
result = model.finalize(status)
All the concepts described above for ModelEngine can be applied to VectorialModelEngine too.
At the start of the simulation, the code of VectorialModelEngine checks that all the steps defined in the configuration file are subclasses of vecrops.VectorialStep. If not, an error is triggered.
The daily details object of the vectorial case is a dictionary, similar to its counterpart in the scalar case. The dictionary contains one item for each output variable: the key is the variable name, the value is a list: the list contains one item per simulated day. Each item of the list is an array, containing one item per simulated location.
Vectorization of models is based on the concept of applying the model algorithm to an input data vector, where each element of the vector represents the data of a different location.
In this way, the model performs the model calculation on all locations simultaneously, achieving much higher performance than using a loop or parallelization method.
The vectorized model uses the numpy library, which allows mathematical and non-mathematical operations to be applied to an array of data.
Whoever launches the model must prepare the input data in the form of an array, having a size equal to the number of locations to be simulated. If the data has other dimensions, in addition to the location dimension, the location dimension must be the first.
For example, an array containing weather data should have dimensions (NUM_LOCATIONS, NUM_DAYS, NUM_WEATHER_VARIABLES).
The results of the simulations are also returned as an array, where the first dimension is that of the different locations.
The implementation of VectorialSteps is characterized by the fact that the algorithm code is written to operate on arrays of values and not on single scalar values.
Therefore, numpy mathematical operations are used (e.g., math.exp on a scalar is replaced with np.exp on a vector), while logical operations must be transformed into array operations.
For example, the following code, where x and y are scalars
- ``if x > 0:
y = 2
- else:
y = 3``
is rewritten as:
y = np.where( x > 0 , 2 , 3 )
where x and y are vectors.
When using the DataLoaders of the ModelLibrary in the scalar case, they return data for all locations to be simulated. The ModelLauncher script loops through all locations and extracts the data for the current location.
When a Model is created in the vectorial version, in the ‘initialize’ method of the model, the user must pass the VectorialModelEngine as ‘modelEngineClass’:
Model.initialize(…, modelEngineClass=VectorialModelEngine)
In the vectorial version, the same DataLoaders are used. Their data is transformed into arrays having the number of locations as the first dimension, and is passed directly to the model.
In the scalar case, before calling the simulation on the single location, the validity of the location is checked by calling the model.isValidLocation method on the location data. Only the locations for which the method returns True are simulated.
In the vectorial case, the model.isValidLocation method returns an array of booleans, having a size equal to the number of locations, which is used to filter the input data.
In pseudocode, this is the difference between the two cases.
Scalar case:
data = dataloder.getdata() #for example data has dimension {DIM_X, DIMY, NUM_DAYS, NUM_VARIABLES)
for each location:
location_data = extract_location_data(data)
isvalid = model.isValidLocation(location_data)
if isvalid then result = model.runModel(location_data)
complete_result.append(result)
Vectorial case:
data = dataloder.getdata() #for example data has dimension {DIM_X, DIMY, NUM_DAYS, NUM_VARIABLES)
data = data.resize(-1, NUM_DAYS,NUM_VARIABLES) # collapse the X and Y dimensions into the locations dimension
isvalid = model.isValidLocation(data) # isvalid is an array of booleans, like [True,False, True,...]
data = data[isvalid] #exclude from the simulations the non valid locations
complete_result = model.runModel(data)
ECroPS model direct run¶
An ECroPS model can be run directly from a python ‘main’ program, by following the example provided in project “EcropsWofostExampleConsole”. Inside the project, there is a “main.py” script that shows the steps to be done. We recommend to have a look at the script and try to execute it to familiarize with the ECroPS engine.
Initialize Wofost ECroPS implementation by reading the workflow file
Get all available run modes, as defined in the workflow file
Define basic input data (year, location data, crop,co2concentrations,…), the soil data and the sowing date-associated
Read the weather data (in this case from a CSV file) into a weather array
Set the Wofost parameter values
For each year to run:
Extract current year weather from weather array
Set the driving variables
For each run mode ( for example potential/water limited):
Initialize the model
Execute the model for the number of weather days
Get the summary model output
Print the simulation results
Please note that if boolean “PrintDailyDetails” is set to True in the script, the engine will print the daily values of the output variables for every simulated day. Otherwise, it is necessary to print explicitly the final output to see the simulation results. The final output contains the values of the output variables as calculated after the last time iteration.
This EcropsWofostExampleConsole console is a sample application to launch an ECroPS simulation. Overall, this code demonstrates how the model can be initialized, executed, and finalized for different run modes, and how the output of the simulation can be obtained and displayed.
The EcropsWofostExampleConsole console performs the following steps:
Initializes the WOFOST model by reading the workflow file and creating a ModelEngine object.
Gets all available run modes, as defined in the workflow file, and sets the ReturnDailyDetails and PrintDailyDetails properties of the ModelEngine object to True. If boolean “PrintDailyDetails” is set to True in the script, the engine will print the daily values of the output variables for every simulated day. Otherwise, it is necessary to print explicitly the final output to see the simulation results. The final output contains the values of the output variables as calculated after the last time iteration.
Defines basic input data for the simulation, including the year, location, crop type, and soil data.
Defines the sowing date for the crop, which is the day of the year (from 1 to 365) on which the crop is planted.
Defines the driving variables dictionary.
Reads the weather data:
Defines the firstYearInWeatherData variable, which specifies the first year in the weather data file.
Calls the ExtractWeather function, passing in the time-dependent variables, and the starting and ending dates of the simulation. The ExtractWeather function returns a subset of the weather data that corresponds to the specified year and dates.
Now, the code can use the extracted weather data to initialize and run the simulation model for the specified year.
After these initial steps, the code can use the ModelEngine object to initialize and run the WOFOST model for different run modes, and obtain the output of the simulation. This code does not contain the entire program, but only the initial setup and configuration of the WOFOST model.
The code performs the following steps for each run mode:
Initializes the model by importing the datetime module, and setting the starting and ending days of the simulation based on the specified year and duration.
Calls the initialize method of the model, passing in the weather data, driving variables, and parameters, as well as the starting and ending days of the simulation. This method returns a status object that contains the initialized state of the model.
Calls the executeStep method of the model repeatedly, passing in the status object and the current run mode. This method simulates the growth of the crop for each day of the simulation.
Calls the finalize method of the model, passing in the status object and the current run mode. This method returns the summary output of the simulation for the current run mode.
Prints the names and values of the output variables for the current run mode.
The provided EcropsWofostExampleConsole contains 4 sample workflow files. They can be used as examples to create other customized workflow.
the simpler workflow is in file ‘WorkflowWofostPhenology.xml’. This workflow includes the steps to simulate the WOFOST phenology model: the development stage and the phenological stage dates are calculated
the workflow in file ‘WorkflowWofostSimple.xml’ includes the steps to simulate Wofost model without taking into account the CO2 concentration. It includes the workflow of the potential simulation and the water limited (rainfed) simulation
the workflow in file ‘WorkflowWofostSimpleWithCo2.xml’, on top of the previous workflow, adds the CO2 effects on transpiration and assimilation
the workflow in file ‘WorkflowWofostCo2Partitioning.xml’, on top of the previous workflows, adds the CO2 effect on partitioning coefficients.
The ECroPS engine allow to build a graph of the workflow defined in a workflow configuration file. The graph shows the interconnections of the steps, and lists the output of a step that are used as inputs of other steps. The graph could be textual or graphical, depending on the graph builder used.
A graph builder is an implementation of the abstract class ecrops.graphbuilders.AbstractGraphBuilder.
An example implementation is provided in this package: class ecrops.graphbuilders.TextualGraphBuilder print the graph of the workflow on the console output.
Example of usage:
workflow_file = WorkflowWofostSimple.xml
runmode = 'Potential'
w = ModelEngine(workflow_file)
graphbuilders = []
graphbuilders.append(TextualGraphBuilder())
w.createModelGraph(runmode, graphbuilders)
Models and DataLoaders¶
As described in the previous paragraphs, it is possible to create a python script that:
reads the input data
runs the model
saves the results
See in the next figure a depiction of the basic usage of ecrops.
Schema of the basic usage of the ECroPS platform: a custom python script calls the ECroPS engine, which in turn calls the model steps. The custom script is responsible of reading the input data and save the simulation results. In the figure, the blue objects are the python classes, which are the generic parts of the platform, whereas the orange parts are the custom implementation for the specific models. The yellow objects are the input and output data, whereas the yellow arrows show the movement of the data between the components.
To create a custom script to launch the simulations requires coding skills by the user and limits the reusability of the written code. Therefore, we have introduced an advanced mode of the platform where the system guides the user in implementing the input reading and output writing components. The user is also guided in the preparation of the simulation and the call of the engine. Since these parts are often common across different models, we promote the reusability of this code. Therefore, we created a ModelLibrary package that contains the logics of reading the input data, preparing the data, calling the model engine and saving the simulation results. These tasks are managed by two distinct set of classes:
The AbstractDataLoader is the prototype for a class that reads the input data of different kind and from different data sources. To make an example, a DataLoader for the Wofost model could read the weather data from a database and the soil data from a textual file.
The AbstractModel is a class that performs these tasks:
Calls the AbstractDataLoader to get the input data
Verifies the input data necessary to run a certain simulation are present
Adjusts the input data according to the model needs (e.g. unit of measure conversion)
Calls the ECroPS engine providing the configuration XML and the adjusted input data
Receives the simulation outputs and saves them in the desired format
Note: since version 1.9.0 of ECroPS package and 1.4.0 of ModelLibrary package, the AbstractDataLoader and the AbstractModel classes were moved from the ModelLibrary package to the ecrops.ModelLibrary package, inside the ecrops code. Starting from the version 1.4.0 of ModelLibrary package, it contains only the realizations of the abstract classes (e.g. the implementations for Wofost and WARM).
AbstractModel and AbstractDataLoader are abstract classes, defined using the package ABC. Their abstract methods are marked as abstract and should be implemented in the derived classes. The AbstractModel is called by a ModelLauncher script. This script has the task of loading the desired implementation of the AbstractModel and to call it for every unit that should be simulated, in a loop cycle. When all the simulation units are processed, the ModelLauncher collects their output and save a cumulative summary output. For running parallel simulations, the ModelLauncher script is replaced by the MPIModelLauncher scripts, that, using the MPI protocol, distributes the simulation units over the resources of a cluster, triggers the parallel execution, and, at the end, collects and saves the cumulative summary output.
See in the next figure the schema of the advanced usage mode of the platform, where we show the elements of the ModelLibrary package. The ECroPS engine is no more called by a custom script, but by an implementation of the AbstractModel class. The rightmost part of the figure, showing the engine behaviour, is identical to that one in the figure above. This is because the behaviour of ECroPS engine does not change between the basic and the advanced usage of the platform.
Schema of the advanced usage of the ECroPS platform. The schema shows a ModelLauncher script (or a MPI ModelLauncher script in case of parallel simulations) calling an AbstractModel implementation. This implementation calls an AbstractDataLoader to load the input data and then calls the ECroPS engine providing the necessary input data. The AbstractModel is also responsible of saving the simulation results. In the figure, the blue objects are the python classes, which are the generic parts of the platform, whereas the orange parts are the custom implementation for the specific models. The yellow objects are the input and output data, whereas the yellow arrows show the movement of the data between the components.
A Model class should be defined as an implementation of AbstractModel class. A DataLoader should be defined as an implementation of AbstractDataLoader class. Below we show the code of the AbstractModel and AbstractDataLoader classes.
class AbstractModel(ABC):
"""Abstract representation of a model implementation"""
def __init__(self, args=None, name=''):
self.Name = name
self.args = args
def getName(self):
"""returns the name of the model"""
return self.Name
def getArgs(self):
"""returns the arguments of the model"""
return self.args
def initialize(self, baseDirectory, log, workflow_file, ReturnDailyDetails, ReturnDekadalDetails, PrintDailyDetails,
PrintDailyDetailsToFile, PrintDailyDetails_OutputFile):
""" Initializes the model:
- Initializes the ModelEngine of ecrops
- Saves in the model properties the base directory, the workflow file, and the arguments related to save and print the model outputs
Arguments
:param baseDirectory: the base directory that contains the input data
:param log: set to True to enable logging inside models
:param workflow_file: name of the workflow XML file
:param ReturnDailyDetails: set True to let Ecrops model to return daily simulation output data. If False, the model will return only summary output data. (better to disable for batch runs).
:param PrintDailyDetails: set True to let Ecrops model to print daily simulation output data in the console output (better to disable for batch runs)
:param ReturnDekadalDetails: set True to let Ecrops model to return 'dekadal' (=10 days) simulation output data. The dekadal configuration returns only the days that are considered dekadal:the 10th , the 20th and the last day of the month.
:param PrintDailyDetailsToFile: set True to save the daily details to a file
:param PrintDailyDetails_OutputFile: the path to save the daily details when PrintDailyDetailsToFile is true
"""
[…]
def getOutputVariables(self):
"""Returns the ordered list of output variables, based on the workflow file loaded.
For each runMode defined in the workflow file, it collects the output variables of the runMode and appends them to the list.
It returns the output variables only of the runModes set to 'ON' in the workflow file.
If a workflow file is not set, it gives an error, so this method should be called after the 'initialize' method """
[…]
@abstractmethod
def runCycle(self, dailyData, dailyDataVariables, locationData, cropLocationData, soilData, otherVariables,
cropLocationDataOrder,
locationDataOrder, soilDataOrder, cropParameters):
"""Executes the simulation cycle of the model, using all the input data retrieved by the data loader. A further elaboration of the input data could be implemented here, before calling the initialize method of the ecrops model.
Typical workflow is:
- prepare/modify the input data
- fill the drivingVariable dictionary by uing the input data
- for each run modes defined in the ecrops workflow:
- call the ModelEngine initialize method
- call N times the ModelEngine executeStep method, where N is the length of the simulation period
- call the ModelEngine finalize method
- concatenate the summary output of the current run mode with the others
-return the summary outputs and the daily details of all the run modes
Arguments
:param dailyData: the array containing the daily data
:param dailyDataVariables: a dictionary, containing for every daily data variable its position in the dailyData array. E.g.: {'temperature_max': 0, 'temperature_min': 1, 'radiation': 2, 'precipitation': 3, 'windspeed': 4, 'e0': 6, 'es0': 7, 'et0': 8, 'temperature_avg': 9}
:param locationData: the array containing the location data
:param cropLocationData: the array containing the crop location specific data
:param soilData: the array containing the soil specific data
:param otherVariables: the dictionary containing all the other variables
:param cropLocationDataOrder: the dictionary containing the crop location specific data order: the order gives the 0 based index position of a variable inside the cropLocationData array
:param locationDataOrder: the dictionary containing the location data order: the order gives the 0 based index position of a variable inside the locationData array
:param soilDataOrder: the dictionary containing the soil data order: the order gives the 0 based index position of a variable inside the soilData array
:param cropParameters: the dictionary containing the non-location specific crop data
Returns a tuple containing the summary output and the daily details dictionary (one entry for each run mode)
"""
pass
@abstractmethod
def getOutputFileName(self, args):
"""
Returns the name of the output file according to some simulation's arguments.
Arguments:
:param args: arguments of the model call
:return: name of the output file
"""
pass
@abstractmethod
def saveFinalOutput(self, data, outputFilePath):
"""save the final results of the simulation
Arguments
:param data: the output data array
:param outputFilePath: the path of the output file to write
"""
pass
@abstractmethod
def isValidLocation(self, dailyData, locationData, locationDataOrder, cropparameters, crop):
"""checks whether the location data provided are valid. Checks are done on daily data and location data.
Returns true if data are valid, False otherwise.
Arguments
:param dailydata: the array containing the daily data
:param locationData: the array containing the location data
:param locationDataOrder: the dictionary containing the daily data order
:param cropparameters: crop parameters
:param crop: crop number
"""
return True
@property
def logger(self):
loggername = "%s.%s" % (self.__class__.__module__,
self.__class__.__name__)
return logging.getLogger(loggername)
@abstractmethod
def prepare_dekadal_output(self, args, timeseries_results):
"""converts timeseries results into dekadal output as a pandas dataframe
:param args: the set of arguments used to start the ModelLauncher
:param timeseries_results: timeseries output from eCrops
:return: a tuple of two objects:
- a DataFrame with results in a custom structure (for example to be converted in a SQLLDR file for CGMS23EUR)
- the string representing the SQLLDR header associated to the dataframe
"""
return pd.DataFrame(), ""
@abstractmethod
def prepare_daily_output(self, args, timeseries_results):
"""converts timeseries results into daily output as a pandas dataframe
:param args: the set of arguments used to start the ModelLauncher
:param timeseries_results: timeseries output from eCrops
:return: a tuple of two objects:
- a DataFrame with results in a custom structure (for example to be converted in a SQLLDR file for CGMS23EUR)
- the string representing the SQLLDR header associated to the dataframe
"""
return pd.DataFrame(), ""
class AbstractDataLoader(ABC):
"""Abstract representation of a data loader class"""
def __init__(self, args=None, Name=''):
self.Name = Name
self.args = args
def getName(self):
"""returns the data loader name"""
return self.Name
def getArgs(self):
"""returns the data loader arguments"""
return self.args
@abstractmethod
def getDailyData(self):
"""returns the input daily data (for example the weather data)"""
pass
@abstractmethod
def getDailyDataVariables(self):
"""returns the names of input daily data (for example the weather data) in a dictionary. The key of the dictionary is the name of the variable, the value is its position in the array returned by getDailyData"""
pass
@abstractmethod
def getLocationData(self):
"""returns an array containing the fixed location data (for example the soil data)"""
pass
@abstractmethod
def getLocationDataOrder(self):
"""returns the list of variables contained in LocationData and their zero-based position index in the array returned by getLocationData()"""
pass
@abstractmethod
def getCropParameters(self):
"""returns the non-location specific crop parameters"""
pass
@abstractmethod
def getCropLocationData(self):
"""returns the location specific crop data and parameters"""
pass
@abstractmethod
def getCropLocationDataOrder(self):
"""returns the list of variables contained in CropLocationData and their zero-based position index in the array returned by getCropLocationData()"""
pass
@abstractmethod
def getSoilData(self):
"""returns the soil specific data"""
return self.soilData
@abstractmethod
def getSoilDimensionSize(self):
"""returns the size of the soil dimension"""
if len(self.soilData) > 0:
return self.soilData.shape[1]
else:
return 1
@abstractmethod
def getSoilDataOrder(self):
"""returns the list of variables contained in getSoilData and their zero-based position index in the array returned by getSoilData()"""
return self.soilDataOrder
@abstractmethod
def getOtherVariables(self):
"""return a dictionary containing other auxiliary variables"""
pass
@property
def logger(self):
loggername = "%s.%s" % (self.__class__.__module__,
self.__class__.__name__)
return logging.getLogger(loggername)
The advantage to create fixed definitions for Models and DataLoaders is that their implementations can be launched by a unique ‘main’ program. The ModelLauncher project contains a “ModelLauncherMain.py” python script that launches an implementation of AbstractModel, and feeds it with the input data provided by an implementation of AbstractDataLoader. This is the schematic workflow implemented in “ModelLauncherMain.py”:
It instantiates the specified DataLoader and calls its methods to retrieve the input data
It instantiates the specified Model
It calls the “initialize” method of the Model
It prepares a list of year/locations to simulate. The invalid locations are excluded. Valid locations are identified by calling the “isValidLocation” method of the Model
For each year/location in the list:
It starts the simulation, calling the Model’s “runCycle” method and passing to it the input data.
It saves the Model’s output into a common output array
It saves the common output array to a file by calling the model’s “saveFinalOutput” method
It saves the daily or decadal results to be uploaded in the database in SQLLDR files, by calling the methods “prepare_daily_output” or “prepare_dekadal_output” of the Model and saving their results as textual files.
Some implementations of Models are provided in the ModelLibrary package: in the context of the CGMS23 operational system these are the models and data loaders distributed, at February 2025:
ECroPSWARMCGMS10kmModel: model to launch the WARM model on the 10km grid
ECroPSWofostCGMS10kmModel: model to launch the Wofost model on the 10km grid
WARMCGMS10kmDataLoader: data loader to load the input data for ECroPSWARMCGMS10kmModel
WofostCGMS10kmDataLoader: data loader to load the input data for ECroPSWofostCGMS10kmModel
Model Library Manual¶
ModelLibrary¶
To run a model these three steps are performed:
read the input data
run the model
save the results
To enhance the code reusability and the modularity of the software, it is useful to encapsulate these tasks in separate classes. For example if the model is the same but the input dataset changes, the user could modify only the part that reads the input data, leaving the rest of the code unchanged.
For this purpose, we created the ModelLibrary package: it contains an abstract definition of what is a “Model” and what is a “Dataloader”. The Model contains the logics to simulate and the logics to save the model outputs. In the ECroPS cotext, the Model initializes the ECroPS engine and calls it by passing to it the input data coming from the Dataloader. The Dataloader fulfils the task of reading the input data.
Since version 1.3.6 of ModelLibrary package, AbstractModel and AbstractDataLoader are abstract classes, defined using the package ABC. Their abstract methods are marked as abstract and should be implemented in the derived classes.
A proper combination of a Model and a Dataloader creates a full working system. See in the next figure the schema of the structure of the packages of the platform. The ECroPS engine can be called directly from a main, or by an implementation of the ModelLibrary. A ‘Main script’ can call directly the ECroPS engine, which in turn calls the model’s steps. A Model Launcher (or an MPI Model Laucher) can instantiate a Data Loader and a Model implementation. The Model Launcher first calls the Dataloader to get the input data, then passes these data to the Model, by using the Model’s ‘runCycle’ method. The Model implementation calls the ECroPS model engine.
A Model class should be defined as an implementation of AbstractModel class. A Dataloader should be defined as an implementation of AbstractDataLoader class. Both AbstractModel and AbstractDataLoader are defined in this package. The package contains also some implementations:
DummyModel and DummyDataLoader are generic ‘dummy’ examples, that run no model and return no data.
EcropsWARMCGMSModel and WARMCGMSDataLoader are the model to run the WARM rice agronomical model, and its dataloader. They are configured to run in the 25km CGMS operational system
EcropsWofostCGMSModel and WofostCGMSDataLoader are the model to run the Wofost agronomical model, and its dataloader. They are configured to run in the 25km CGMS operational system
EcropsWofostCGMSMedoidModel and WofostCGMSMedoidsDataLoader are the model to run the Wofost agronomical model, and its dataloader. They are configured to run in the new 10km CGMS operational system, using the new soil data/soil water model
EcropsWofostPesetaModel and WofostPesetaDataLoader are the model to run the Wofost agronomical model, and its dataloader in the context of the Peseta project. They are configured to run in the 25km CGMS operational system, but using climate scenario weather data, and using only one soil per cell
The default model engine instantiate by AbstractModel is ecrops.ModelEngine, but, since version 1.3.7, it is possible to provide in the AbstractModel class an alternative engine, by using the modelEngineClass argument of the initialize method.
- ``def initialize(self, baseDirectory, log, workflow_file, ReturnDailyDetails, ReturnDekadalDetails, PrintDailyDetails,
PrintDailyDetailsToFile, PrintDailyDetails_OutputFile, modelEngineClass=ModelEngine)``
Model Launcher¶
The couple Model-Dataloader can be launched by a proper ‘main’ program that is the same for each Model and Dataloader selected: the ModelLauncher project contains a “ModelLauncherMain.py” python script that launches an implementation of AbstractModel, and feeds it with the input data provided by an implementation of AbstractDataLoader. This is the schematic workflow implemented in “ModelLauncherMain.py”:
instantiate a data loader and calls its methods to retrieve the input database
instantiate the model
call the initialize method of the model
prepare a list of year/locations to simulate. The invalid locations are excluded. Valid locations are identified by using isValidLocation method of the model
for each year/location in the list:
calls the model, passing the input data as arguments of runCycle method
saves the model output into a common output array
saves the common output array to a file by calling the model’s saveFinalOutput method
Workflow graphs¶
In this section, we show the workflow graph of some of the model configurations.